If you are interested in analyzing the deleted tweets of politicians, my R package PolitwoopsR is on GitHub. Also see this R script examining deleted tweets. This post is from 2015, and the Politwoops and Twitter APIs it relies on have changed since. If you were brought here by a Dean Richards Twitter bot, read this.
A few days ago I published a post with interactive visualizations of the deleted tweets of US politicians. Those tweets were collected by the Politwoops project (now offline; the link opens an archived copy), maintained in the US by the Sunlight Foundation. I promised to post the R code that grabs the tweet and politician data, cleans it, and generates the charts. As I started organizing the code, it seemed easy to take the next step and create a package that handles the data collection.
The PolitwoopsR package includes functions that extract tweet data from most of the country-specific Politwoops projects. It also provides a way to scrape politician information, though that only works with the US project. I have tested it with only a few countries (US, UK, Argentina), so you may still run into bugs with countries that use a non-standard JSON format or URL form.
I have also posted an example script in this GitHub gist that collects and cleans this type of data. It includes the following steps:
- Getting the data using the PolitwoopsR package
- Expanding the short URLs in tweets and extracting domains
- Detecting the language of the tweets
- Cleaning the tweet text, stemming and stem completion, top terms
- Top hashtags by political party
- Tweet sentiment and polarity
- Collecting additional data from Twitter with the twitteR package
- Downloading the politicians’ Twitter profile images to use in visualizations
PolitwoopsR examples
Here are a few quick examples of what the package can do. Before getting started, install PolitwoopsR from GitHub using the devtools package.
install.packages('devtools')
library(devtools)
install_github('kateto/PolitwoopsR')
library(PolitwoopsR)Politwoops provides tweet data as paginated JSON (usually 10 or 20 tweets per page). To get a data frame with that data, use get_pw_tweets(). It takes four arguments: start.page (defaults to the first page), end.page (defaults to the last page available), domain (defaults to the US domain), and json.ext (defaults to "/index.json?page=", which is where you find the JSON for the US and UK projects). Getting all the pages (over 750 for the US at the time) may take a while.
# Get tweet JSONs - all pages from the US site:
pw.df <- get_pw_tweets()
# Get the first 5 pages only from the UK Politwoops tweets:
pw.df <- get_pw_tweets(1, 5, "politwoops.co.uk")
# Get the first 5 pages from Argentina's Politwoops tweets:
tweet.df <- get_pw_tweets(1, 5, "www.politwoops.com/g/Argentina", ".json?page=")The politician data is not available in a convenient format, but can be scraped from HTML tables on the US site with get_pw_pol(). Since it only works with the US site, it takes just two arguments: start.page (defaults to 1) and end.page (defaults to last).
# Get all politician data:
pol.inf <- get_pw_pol()The function merge_pw() does a left outer join of the tweet and politician data frames, merging by Twitter handle. In other words, it adds columns to the tweet data frame with information about the politician who wrote each tweet.
# Combine tweet and politician data:
pw.dfs <- merge_pw(pw.df, pol.inf)Helper functions in the package also include short URL expansion and domain extraction:
# Expand URLs:
pw.dfs$url.decoded <- url_expand(pw.df)
# Extract domains from the URLs:
pw.dfs$url.domain <- url_domain(pw.dfs$url.decoded)The Dean Richards bot tale
After putting the PolitwoopsR package on GitHub, I sent out a tweet to let #rstats folks know about it. It was retweeted by Pew’s Claudia Deane, who was at the time getting a lot of unfortunate attention from an army of Russian bots.
All of the bots used the name, image, and profile info of WGN reporter Dean Richards (whose real account is verified), and their usernames were “c_deane” followed by a random string of letters. The bots would furiously repost (not retweet, but copy and paste without credit) anything they found on Claudia’s timeline. That included my tweet, which spent days being rebroadcast by bots several times an hour.
I alerted Twitter, though it’s hard to say whether that had any effect; the situation didn’t fit neatly in their spam, harassment, or copyright categories, especially since it involved many bot accounts rather than a single offender. The story has no real conclusion, except that (as science fiction has taught us) robot armies are tough to beat. So if you landed here because of a Dean Richards bot tweet, I hope you found this useful anyway. And please say hi to our robotic overlords for me.